Vehicle control apparatus, vehicle, vehicle control method, and storage medium

ABSTRACT

There is provided a vehicle control apparatus that controls automated driving of a vehicle. The apparatus includes an extraction unit configured to extract an object existing around the vehicle from a scene image representing a peripheral status of the vehicle, and a control unit configured to calculate a moving locus of the object and a moving locus of the vehicle for a predetermined time from time when the scene image is acquired, and generate a moving locus by correcting the moving locus of the object based on the moving locus of the vehicle.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to a vehicle control apparatus, a vehicle,a vehicle control method, and a storage medium and, more particularly,to a vehicle control technique for controlling an automated driving car.

Description of the Related Art

Automated driving of a vehicle is implemented by recognizing theperipheral environment of the vehicle, determining, based on therecognition result, a moving locus along which the vehicle is to move,and performing steering control for causing the vehicle to travel inaccordance with the moving locus. When determining the moving locus, thepositions of a moving object and stationary object on a road or aroundthe vehicle are specified while predicting the future position of themoving object, thereby determining, in accordance with the specifyingand prediction results, a position at which the vehicle is to exist ateach future time. For example, the moving locus of the vehicle isdetermined so that the vehicle exists at each time in a region where noobject exists.

Japanese Patent Laid-Open No. 2011-081736 discloses an arrangement inwhich when detecting a pedestrian based on an image obtained byperforming image capturing, a pedestrian outside a preset range isexcluded from candidates.

However, in the arrangement disclosed in Japanese Patent Laid-Open No.2011-081736, for example, since a moving object (object) existing at along distance is excluded from prediction target candidates, if acontrol target vehicle takes a moving locus to get closer to the objectwith the lapse of time, it may be impossible to predict the motion ofthe object as a prediction target or obtain, based on the moving locusof the control target vehicle, the moving locus of the object whosepredicted motion has been corrected.

SUMMARY OF THE INVENTION

The present invention has been made to solve at least the above problem,and provides a vehicle control technique capable of obtaining, based onthe moving locus of a control target vehicle, the moving locus of anobject whose predicted motion has been corrected.

According to one aspect of the present invention, there is provided avehicle control apparatus that controls automated driving of a vehiclebased on a generated moving locus, comprising: an extraction unitconfigured to extract an object existing around the vehicle from a sceneimage representing a peripheral status of the vehicle; and a controlunit configured to calculate a moving locus of the object and a movinglocus of the vehicle for a predetermined time from time when the sceneimage is acquired, and generate a moving locus by correcting the movinglocus of the object based on the moving locus of the vehicle.

According to the present invention, it is possible to obtain, based onthe moving locus of a control target vehicle, the moving locus of anobject whose predicted motion has been corrected.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a block diagram showing the basic arrangement of a vehiclecontrol apparatus;

FIG. 1B is a view showing an example of the arrangement of controlblocks for controlling a vehicle;

FIGS. 2A to 2C are views each showing an outline of processing inautomated driving control;

FIG. 3 is a view schematically showing the relationship between a timeaxis (ordinate) when a self-vehicle travels and a time axis (abscissa)used for calculation when obtaining a moving locus;

FIG. 4 is a flowchart for explaining the procedure of processing in thefirst automated driving mode;

FIG. 5 is a view exemplifying a scene image acquired at given time;

FIGS. 6A to 6C are views each for exemplarily explaining determinationof the moving locus of the self-vehicle;

FIG. 7 is a view schematically showing the relationship between a timeaxis (ordinate) when a self-vehicle travels and a time axis (abscissa)used for calculation when obtaining a moving locus;

FIG. 8 is a flowchart for explaining the procedure of processing in thesecond automated driving mode;

FIG. 9 is a view schematically showing the relationship between a timeaxis (ordinate) when a self-vehicle travels and a time axis (abscissa)used for calculation when obtaining a moving locus;

FIG. 10 is a flowchart for explaining the procedure of processing in thethird automated driving mode; and

FIG. 11 is a view exemplifying a scene image obtained at given time.

DESCRIPTION OF THE EMBODIMENTS

Embodiments of the present invention will be described below withreference to the accompanying drawings. The constituent elementsdescribed in the embodiments are merely examples. The present inventionis not limited by the following embodiments.

(Arrangement of Vehicle Control Apparatus)

FIG. 1A is a block diagram showing the basic arrangement of a vehiclecontrol apparatus 100 that controls automated driving of a vehicle. Thevehicle control apparatus 100 includes a sensor S, a camera CAM, and acomputer COM. The sensor S includes, for example, a radar S1 and a lidarS2. The computer COM includes a CPU C1 that controls execution ofprocessing, a memory C2, and an interface (I/F) C3 with an externaldevice. The sensor S and the camera CAM acquire external information ofthe vehicle, and input it to the computer COM. A vehicle mounted withthe computer COM will also be referred to as a self-vehicle hereinafter,and a two- or four-wheeled vehicle existing around the self-vehicle willalso be referred to as another vehicle hereinafter.

The computer COM performs image processing for the information inputfrom the sensor S and the camera CAM, thereby extracting objectsincluded in an image (scene image) at given time when the externalinformation is acquired. The objects include, for example, a staticobject (for example, a stationary object such as a road structureincluding a lane, traffic light, curb, traffic sign, or guardrail) thatdoes not move with the lapse of time, and a dynamic object (for example,a moving object such as another vehicle or a pedestrian) that moves withthe lapse of time.

For example, at given time T0, the computer COM extracts, from an image(scene image) acquired by the sensor S and the camera CAM, objectsincluded in the scene image, and analyzes objects arranged around theself-vehicle (scene understanding processing).

The computer COM specifies the objects extracted from the scene image bylabeling them, and performs, for each object, prediction calculation ofthe motion (moving locus) of the object for a predetermined time TS fromgiven time to the future (prediction processing).

Based on the predicted moving locus of each object, the computer COMcalculates the moving locus of the self-vehicle, indicating that theself-vehicle is to move to a specific position until specific time(action planning processing). When generating the moving locus of theself-vehicle, the computer COM can determine the moving locus of theself-vehicle in the acquired scene image in consideration of, forexample, data of a combination of traveling by a skilled driver and theperipheral status of the self-vehicle, that is detected at this time.Practical processing contents will be described in detail later.

In this embodiment, the computer COM can control automated driving byswitching among a plurality of automated driving modes. The plurality ofautomated driving modes include an automated driving mode (firstautomated driving mode) which gives priority to the calculation time andin which the action plan of the self-vehicle is calculated within ashort time by performing the scene understanding, prediction, and actionplanning processes by one processing sequence (one sequence), anautomated driving mode (second automated driving mode) which givespriority to the accuracy and in which the action plan of theself-vehicle is calculated at high accuracy by performing a loopoperation of repeating the prediction and action planning processes at apredetermined operation period, and an automated driving mode (thirdautomated driving mode) in which an action plan obtained in a sceneimage at given time T1 is calculated using an action plan obtained froma scene image at time T0 before time T1, thereby ensuring the accuracyof the action plan while reducing the processing load.

The automated driving mode can be switched in accordance with, forexample, setting of a driver. Alternatively, the computer COM cancontrol automated driving by automatically switching the automateddriving mode in accordance with the external status surrounding theself-vehicle, such as the type of road (highway, ordinary road, or thelike) on which the vehicle travels, information (for example, a fewpedestrians and many other vehicles or many pedestrians and many othervehicles) of the objects extracted from the scene image, a time period,like the daytime or nighttime, during which the vehicle travels, and theweather.

If the vehicle control apparatus 100 shown in FIG. 1A is mounted on thevehicle, the computer COM may be arranged in an ECU of an imageprocessing system or an ECU of a recognition processing system, thatprocesses information of the sensor S and the camera CAM, an ECU in acontrol unit that controls driving of the vehicle, or an ECU forautomated driving. Alternatively, as shown in FIG. 1B to be describedbelow, functions may be distributed to the plurality of ECUsconstituting the vehicle control apparatus 100, such as an ECU for thesensor S, an ECU for the camera, and an ECU for automated driving.

FIG. 1B is a view showing an example of the arrangement of the controlblocks of the vehicle control apparatus 100 for controlling a vehicle 1.Referring to FIG. 1B, an outline of the vehicle 1 is shown in a planview and a side view. As an example, the vehicle 1 is a sedan-typefour-wheeled vehicle.

A control unit 2 shown in FIG. 1B controls each unit of the vehicle 1.The control unit 2 includes a plurality of ECUs 20 to 29 communicablyconnected by an in-vehicle network. Each ECU (Engine Control Unit)includes a processor represented by a CPU (Central Processing Unit), astorage device such as a semiconductor memory, and an interface with anexternal device. The storage device stores programs to be executed bythe processor, data to be used by the processor for processing, and thelike. Each ECU may include a plurality of processors, storage devices,and interfaces.

The functions and the like provided by the ECUs 20 to 29 will bedescribed below. Note that the number of ECUs and the provided functionscan appropriately be designed for the vehicle 1, and they can besubdivided or integrated as compared to this embodiment.

The ECU 20 executes control associated with automated driving of thevehicle 1. In automated driving, at least one of steering andacceleration/deceleration of the vehicle 1 is automatically controlled.Processing for practical control associated with automated driving willbe described in detail later.

The ECU 21 controls an electric power steering device 3. The electricpower steering device 3 includes a mechanism that steers front wheels inaccordance with a driving operation (steering operation) of the driveron a steering wheel 31. In addition, the electric power steering device3 includes a motor that generates a driving force to assist the steeringoperation or automatically steer the front wheels, and a sensor thatdetects the steering angle. If the driving state of the vehicle 1 isautomated driving, the ECU 21 automatically controls the electric powersteering device 3 in correspondence with an instruction from the ECU 20and controls the traveling direction of the vehicle 1.

The ECUs 22 and 23 perform control of detection units 41 to 43 thatdetect the peripheral status of the vehicle and information processingof detection results. The detection unit 41 is, for example, a camera(to be sometimes referred to as the camera 41 hereinafter) that capturesthe front side of the vehicle 1. In this embodiment, two cameras 41 areprovided on the roof front of the vehicle 1. When images captured by thecameras 41 are analyzed (image processing), the contour of a target or adivision line (a white line or the like) of a lane on a road can beextracted.

The detection unit 42 is, for example, a lidar (laser radar) (to besometimes referred to as the lidar 42 hereinafter), and detects a targetaround the vehicle 1 or measures the distance to a target. In thisembodiment, a plurality of lidars 42 are provided around the vehicle. Inthe example shown in FIG. 1B, five lidars 42 are provided; one at eachcorner of the front portion of the vehicle 1, one at the center of therear portion, and one on each side of the rear portion. The detectionunit 43 is, for example, a millimeter wave radar (to be sometimesreferred to as the radar 43 hereinafter), and detects a target aroundthe vehicle 1 or measures the distance to a target. In this embodiment,a plurality of radars 43 are provided around the vehicle. In the exampleshown in FIG. 1B, five radars 43 are provided; one at the center of thefront portion of the vehicle 1, one at each corner of the front portion,and one at each corner of the rear portion.

The ECU 22 performs control of one camera 41 and each lidar 42 andinformation processing of detection results. The ECU 23 performs controlof the other camera 41 and each radar 43 and information processing ofdetection results. Since two sets of devices that detect the peripheralstatus of the vehicle are provided, the reliability of detection resultscan be improved. In addition, since detection units of different typessuch as cameras, lidars, and radars are provided, the peripheralenvironment of the vehicle can be analyzed multilaterally. Note that theECUs 22 and 23 may be integrated into one ECU.

The ECU 24 performs control of a gyro sensor 5, a GPS sensor 24 b, and acommunication device 24 c and information processing of detectionresults or communication results. The gyro sensor 5 detects a rotarymotion of the vehicle 1. The course of the vehicle 1 can be determinedbased on the detection result of the gyro sensor 5, the wheel speed, orthe like. The GPS sensor 24 b detects the current position of thevehicle 1. The communication device 24 c performs wireless communicationwith a server that provides map information or traffic information andacquires these pieces of information. The ECU 24 can access a mapinformation database 24 a formed in the storage device. The ECU 24searches for a route from the current position to the destination. Thedatabase 24 a can be arranged on the network, and the communicationdevice 24 c can acquire the information by accessing the database 24 aon the network.

The ECU 25 includes a communication device 25 a for inter-vehiclecommunication. The communication device 25 a performs wirelesscommunication with another vehicle on the periphery and performsinformation exchange between the vehicles.

The ECU 26 controls a power plant 6. The power plant 6 is a mechanismthat outputs a driving force to rotate the driving wheels of the vehicle1 and includes, for example, an engine and a transmission. The ECU 26,for example, controls the output of the engine in correspondence with adriving operation (accelerator operation or acceleration operation) ofthe driver detected by an operation detection sensor 7 a provided on anaccelerator pedal 7A, or switches the gear ratio of the transmissionbased on information such as a vehicle speed detected by a vehicle speedsensor 7 c. If the driving state of the vehicle 1 is automated driving,the ECU 26 automatically controls the power plant 6 in correspondencewith an instruction from the ECU 20 and controls theacceleration/deceleration of the vehicle 1.

The ECU 27 controls lighting devices (headlights, taillights, and thelike) including direction indicators 8. In the example shown in FIG. 1B,the direction indicators 8 are provided in the front portion, doormirrors, and the rear portion of the vehicle 1.

The ECU 28 controls an input/output device 9. The input/output device 9outputs information to the driver and accepts input of information fromthe driver. A voice output device 91 notifies the driver of theinformation by a voice. A display device 92 notifies the driver ofinformation by displaying an image. The display device 92 is arrangedon, for example, the surface of the driver's seat and constitutes aninstrument panel or the like. Note that although a voice and displayhave been exemplified here, the driver may be notified of informationusing a vibration or light. Alternatively, the driver may be notified ofinformation by a combination of some of the voice, display, vibration,and light. Furthermore, the combination or the notification form may bechanged in accordance with the level (for example, the degree ofurgency) of information of which the driver is to be notified.

An input device 93 is a switch group that is arranged at a positionwhere the driver can perform an operation and used to issue aninstruction to the vehicle 1, and may also include a voice input device.

The ECU 29 controls a brake device 10 and a parking brake (not shown).The brake device 10 is, for example, a disc brake device which isprovided for each wheel of the vehicle 1 and decelerates or stops thevehicle 1 by applying a resistance to the rotation of the wheel. The ECU29, for example, controls the operation of the brake device 10 incorrespondence with a driving operation (brake operation) of the driverdetected by an operation detection sensor 7 b provided on a brake pedal7B. If the driving state of the vehicle 1 is automated driving, the ECU29 automatically controls the brake device 10 in correspondence with aninstruction from the ECU 20 and controls deceleration and stop of thevehicle 1. The brake device 10 or the parking brake can also be operatedto maintain the stop state of the vehicle 1. In addition, if thetransmission of the power plant 6 includes a parking lock mechanism, itcan be operated to maintain the stop state of the vehicle 1.

(Outline of Processing in Automated Driving Control)

Automated driving control generally includes three steps, as shown inFIGS. 2A to 2C. That is, automated driving control includes, as thefirst step, scene understanding processing of recognizing externalinformation of the periphery of the vehicle 1 (self-vehicle), as thesecond step, prediction processing of predicting a moving locus alongwhich an object extracted from a scene image by the scene understandingprocessing, and as the third step, action planning processing ofdetermining, based on the result of the prediction processing, a movinglocus that should be taken by the vehicle 1 (self-vehicle).

In the following description, the first automated driving mode (FIG. 2A)which gives priority to the calculation time and in which the sceneunderstanding, prediction, and action planning processes are performedby one sequence, the second automated driving mode (FIG. 2B) which givespriority to the accuracy and in which the action plan of the vehicle 1(self-vehicle) is calculated at high accuracy by performing a loopoperation of repeating the prediction and action planning processesamong the scene understanding, prediction, and action planning processesat a predetermined operation period, and the third automated drivingmode (FIG. 2C) in which an action plan obtained in a scene image atgiven time T1 is calculated using an action plan obtained in a sceneimage at time T0 before time T1 will be explained in detail below. Amongthe automated driving modes, the scene understanding processing isbasically the same but the prediction and action planning processes havedifferent characteristics.

First Embodiment: First Automated Driving Mode

In this embodiment, an ECU 20 executes control associated with automateddriving of a vehicle 1. The ECU 20 controls automated driving of thevehicle 1 as a control target based on a calculated moving locus of theself-vehicle. FIG. 2A is a view showing an outline of a processingsequence corresponding to the first automated driving mode. In the firstautomated driving mode, the ECU 20 performs scene understanding,prediction, and action planning processes by one sequence. In the firstautomated driving mode, unlike the second automated driving mode (to bedescribed later), no loop operation of repeating the prediction andaction planning processes among the scene understanding, prediction, andaction planning processes is performed. Therefore, as compared to thesecond automated driving mode, it is possible to reduce the calculationamount, and perform the processes from the scene understandingprocessing to the action planning processing within a shorter time.

FIG. 3 is a view schematically showing the relationship between a timeaxis (ordinate) representing actual time when the vehicle 1(self-vehicle) travels and a calculation time axis (abscissa) to be usedfor calculation when obtaining the moving locus of the dynamic object(for example, another vehicle 205 shown in FIG. 5) and that of thevehicle 1 (self-vehicle). For example, at time T0, based on information(external information) acquired by detection units (camera 41, lidar 42,and radar 43), pieces of information of a static object and dynamicobject existing around the vehicle 1 (self-vehicle) are obtained as animage representing the peripheral status of the vehicle (a scene imagerepresenting one scene), as shown in, for example, FIG. 5.

In the first automated driving mode, the ECU 20 time-serially calculatesthe moving locus of an object for a predetermined time (TS: several sec,for example) from time (for example, time T0) when the scene image isacquired, and time-serially calculates the moving locus of the vehicle 1(self-vehicle) based on the position of the moving locus of the objectfor each time.

For example, the ECU 20 predicts a moving locus along which a dynamicobject such as a pedestrian or another vehicle that moves with the lapseof time moves for the predetermined time TS from time TO when the sceneimage is acquired. The moving locus is predicted by time-seriallypredicting the position of the object for the predetermined time TS ateach time t1, t2, t3, . . . obtained by dividing the predetermined timeTS. Then, based on the moving locus of the object for the predeterminedtime, the ECU 20 time-serially calculates the moving locus of thevehicle 1 (self-vehicle) for the predetermined time TS at each time t1,t2, t3, . . . .

By the series of scene understanding, prediction, and action planningprocesses (processes of one sequence), a calculation result (predictionresult) of the moving locus of the object and a calculation result(action plan) of the moving locus of the vehicle 1 (self-vehicle) basedon the prediction result of the moving locus of the object are obtained.The predetermined time TS is an arbitrarily settable time, and a shorterpredetermined time TS can be set.

FIG. 4 is a flowchart for explaining the procedure of processing in thefirst automated driving mode. When, for example, a driver instructssetting of a destination and automated driving, the ECU 20 automaticallycontrols traveling of the vehicle 1 to the destination in accordancewith a guidance route found by an ECU 24.

At the time of automated driving control, the ECU 20 acquiresinformation (external information) concerning the peripheral status ofthe vehicle 1 from ECUs 22 and 23 (step S10). By performing theprocessing in step S10, an image (scene image) representing theperipheral status of the vehicle is acquired based on the externalinformation at time T0, as shown in, for example, FIG. 5.

The ECU 20 extracts objects (a pedestrian 204, the other vehicle 205,and an obstacle 217) included in the scene image at time (for example,time T0) when the external information is acquired (step S11). The ECU20 extracts, from the scene image, a static object (stationary object)that does not move with the lapse of time and a dynamic object (movingobject) that moves with the lapse of time, and specifies the arrangementof the extracted objects in the scene image. The ECU 20 fixes theposition of the vehicle 1 (self-vehicle) in the scene image as theorigin of a coordinate system, specifying the positions of the objects.

With respect to the dynamic object (moving object) among the extractedobjects, the ECU 20 time-serially calculates and predicts the motion(moving locus) of the object for the predetermined time TS from time T0,when the external information is acquired, toward the future (step S12).The ECU 20 time-serially predicts the position of the object for thepredetermined time TS at each time t1, t2, t3, . . . obtained bydividing the predetermined time TS.

Based on the time-serially calculated moving locus of the dynamic objectfor the predetermined time, the ECU 20 time-serially calculates themoving locus of the vehicle 1 (self-vehicle) for the predetermined time(step S13). Based on the result of time-serially calculating the movinglocus of the dynamic object in the scene image acquired at time T0, theECU 20 calculates the time-series moving locus of the vehicle 1(self-vehicle) in consideration of traveling data by a skilled driverand the positions of the objects existing around the vehicle 1(self-vehicle) in the scene image. The ECU 20 time-serially calculatesthe position of the vehicle 1 (self-vehicle) for the predetermined timeTS at each time t1, t2, t3, . . . obtained by dividing the predeterminedtime TS.

Based on the calculation result of the moving locus of the object ateach time t1, t2, t3, . . . , the ECU 20 determines a moving locus alongwhich the vehicle 1 (self-vehicle) should move at each time (step S14).Based on the operation result in step S13, the ECU 20 determines themoving locus of the vehicle 1 (self-vehicle). The above processing makesit possible to time-serially obtain the moving locus of the vehicle 1(self-vehicle) corresponding to the time-series moving locus of theobject.

Detailed contents of the processing in the first automated driving modewill be described next with reference to FIGS. 5 and 6A to 6C. FIG. 5 isa view exemplifying the scene image based on the external informationacquired by the detection units (camera 41, lidar 42, and radar 43) atgiven time (for example, time t0). The pedestrian 204 and the othervehicle 205 are exemplified as objects existing around the vehicle 1(self-vehicle) in the scene image.

(Practical Example of Scene Understanding)

When the processing shown in FIG. 4 starts, the ECUs 22 and 23 processthe information concerning the peripheral status of the vehicle 1(self-vehicle) based on the information acquired by the detection units(camera 41, lidar 42, and radar 43). The ECU 20 acquires the information(external information) concerning the peripheral status of the vehicle 1(self-vehicle) from the ECUs 22 and 23. Based on the acquired externalinformation, the ECU 20 acquires an image obtained when viewing thevehicle 1 and its peripheral status from above and representing theperipheral status of the vehicle (a scene image representing one scene),on which the objects around the vehicle 1 are mapped, as shown in, forexample, FIG. 5. The above processing corresponds to step S10 of FIG. 4.The ECU 20 fixes the position of the vehicle 1 (self-vehicle) in thescene image as the origin of a coordinate system, executing thefollowing operation processing.

The ECU 20 extracts the static object and dynamic object included in thescene image at time t0. Referring to FIG. 5, the vehicle 1(self-vehicle) travels on the left lane (from the lower side to theupper side on the drawing of FIG. 5) divided by a center line 203 in arange where a vehicle can travel, that is indicated by division lines201 and 202 (for example, outside lines and lines corresponding toshoulders). For example, a guardrail, curb, and the like (not shown) areextracted as stationary objects.

In the traveling direction of the vehicle 1 (self-vehicle), theexistence of the pedestrian 204 on the side of a sidewalk located on theleft side with respect to the division line 202 and the existence of theother vehicle 205 on the side of an opposite lane are extracted asdynamic objects. In addition, the existence of the obstacle 217 isextracted as a stationary object on the division line 201 on the side ofthe opposite lane.

Note that FIG. 5 shows examples of the other vehicle 205 and thepedestrian 204 as dynamic objects in the scene image at given time T0.However, the present invention is not limited to this. For example,another traffic participant such as a bicycle or two-wheeled vehicle canexist on the road or around the vehicle 1 (self-vehicle). It can beassumed that a plurality of other vehicles or a plurality of pedestriansexist. The above processing corresponds to step S11 of FIG. 4.

(Practical Example of Prediction Processing)

Upon acquiring the information (external information) concerning theperipheral status of the vehicle 1 from the ECUs 22 and 23, the ECU 20predicts a moving locus of each of the objects (moving objects such asthe pedestrian 204 and the other vehicle 205) based on the externalinformation. FIG. 5 exemplifies the moving locus (211 to 213 or 505 to507) obtained by predicting the motion of each object.

Referring to FIG. 5, a range within which the pedestrian 204 ispredicted to exist in the future is represented by each of the one-dotdashed line 211, dotted line 212, and two-dot dashed line 213surrounding the pedestrian 204. For example, the one-dot dashed line 211indicates the predicted range of the pedestrian 204 at time t1. Therange indicated by the dotted line 212 is a range within which thepedestrian 204 is predicted to exist at time t2 after time t1.Similarly, the range indicated by the two-dot dashed line 213 is a rangewithin which the pedestrian 204 is predicted to exist at time t3 aftertime t2.

For prediction of the movement of the pedestrian, the direction of theface or that of the body is determined by a combination of parts such aseyes, nose, arms, and feet of the pedestrian 204. Since, in accordancewith the direction of the face of the pedestrian 204, it is predictedthat the pedestrian 204 is to move in the direction of the face, apredicted direction in which the pedestrian 204 is to move can be thedirection of the face (for example, the direction of an arrow 214). Inconsideration of the direction of the face, the direction of the body,or the like of the pedestrian 204, the ECU 20 predicts the moving locusof the pedestrian 204 for the predetermined time TS at each time t1, t2,t3, . . . obtained by dividing the predetermined time TS.

Referring to FIG. 5, to avoid the obstacle 217, the other vehicle 205travels in a state in which it is offset toward the center line 203. Inconsideration of the presence/absence of an obstacle and the status ofthe lane on which the other vehicle 205 travels, the ECU 20 predicts themoving locus of the other vehicle 205 for the predetermined time TS ateach time t1, t2, t3, . . . obtained by dividing the predetermined timeTS. The above processing corresponds to the processing in step S12 ofFIG. 4. The moving locus 505 indicated by two-dot dashed line representsthe moving locus of the other vehicle 205 obtained by the processing inthis step.

(Practical Example of Action Planning)

As processing corresponding to step S13 of FIG. 4, the ECU 20calculates, as the moving locus of the vehicle 1 (self-vehicle), theposition of the vehicle 1 at each time t1, t2, t3, . . . based on themoving locus 505 of the object. Then, the ECU 20 determines, as themoving locus of the vehicle 1 (self-vehicle), the moving locuscalculated in step S13 (step S14). Referring to FIG. 5, a moving locus501 indicated by a two-dot dashed line exemplifies the moving locus ofthe vehicle 1 (self-vehicle) calculated by the processes in steps S13and S14 of FIG. 4. In accordance with the determined moving locus 501,the ECU 20 instructs the ECUs 21, 26, and 29 to control steering,driving, and braking of the vehicle 1.

FIGS. 6A to 6C are views each for exemplarily explaining calculation anddetermination of the moving locus of the vehicle 1 (self-vehicle).Referring to FIG. 6A, each of curves 601 and 602 indicates thedistribution of prediction values at which the object (the pedestrian204 or the other vehicle 205) is predicted to exist at a predeterminedposition (a position between P and P′ in FIG. 5) in the scene image. Thecurve 601 indicates the distribution of prediction values concerning thepedestrian 204 shown in FIG. 5. The curve 602 indicates the distributionof prediction values concerning the other vehicle 205 shown in FIG. 5.

A rectangle 604 indicates the distribution of prediction valuesconcerning a stationary object such as a curb (not shown) or theobstacle 217 as the static object (stationary object). As for the staticobject (stationary object), the object remains at the position withoutmoving, and thus the distribution of the prediction values is arectangular distribution such that a large value is obtained at theposition and zero or sufficiently small values are obtained at theremaining positions.

A region between the curves 601 and 602 is a region where no objectexists on the lane on which the vehicle 1 (self-vehicle) travels, and aregion where the vehicle 1 (self-vehicle) can travel. A curve 603indicates the distribution of traveling patterns by the skilled driverin the travelable region. A peak M1 of the curve 603 indicates atraveling position that is selected by the skilled driver at the highestprobability in the travelable region.

The ECU 20 specifies a region (travelable region) where no object existsat each time (for example, t1, t2, t3, . . . , TS) shown in FIG. 3, andcalculates, based on the distribution of the traveling patterns by theskilled driver, a moving locus so as to move the vehicle 1 to theposition of the peak M1 of the curve 603. The thus calculated movinglocus is set as the moving locus 501 of the vehicle 1 (self-vehicle)obtained in steps S13 and S14 of FIG. 4.

Note that the “skilled driver” is, for example, a professional driver oran excellent driver who has no accidents or traffic violations. Inaddition, traveling data of vehicles by a number of drivers may becollected, and traveling data that satisfy a predetermined criterionsuch as a criterion that no sudden lane change, sudden start, suddenbraking, or sudden steering is performed or a criterion that thetraveling speed is stable may be extracted from the collected travelingdata and reflected, as traveling data of a skilled driver, on automateddriving control by the ECU 20.

In the first automated driving mode, the moving locus of the vehicle 1(self-vehicle) is time-serially calculated in correspondence with thetime-series moving locus of each object. This makes it possible totime-serially determine the moving locus of the vehicle 1 (self-vehicle)so the vehicle 1 does not interfere with the objects (the pedestrian 204and the other vehicle 205) included in the scene image.

Second Embodiment: Second Automated Driving Mode

In this embodiment, an ECU 20 executes control associated with automateddriving of a vehicle 1. The ECU 20 controls automated driving of thevehicle 1 as a control target based on a calculated moving locus. FIG.2B is a view showing an outline of a processing sequence correspondingto the second automated driving mode. In the second automated drivingmode, the ECU 20 performs a loop operation of repeating prediction andaction planning processes among scene understanding, prediction, andaction planning processes at a predetermined operation period,calculating the action plan of the vehicle 1 (self-vehicle) at higheraccuracy.

FIG. 7 is a view schematically showing the relationship between a timeaxis (ordinate) representing actual time when the vehicle 1(self-vehicle) travels and a calculation time axis (abscissa) to be usedfor calculation when obtaining the moving locus of the dynamic object(for example, another vehicle 205 shown in FIG. 5) and that of thevehicle 1 (self-vehicle). For example, at time T0, based on information(external information) acquired by detection units (camera 41, lidar 42,and radar 43), pieces of information of a static object and dynamicobject existing around the vehicle 1 (self-vehicle) are obtained as animage representing the peripheral status of the vehicle (a scene imagerepresenting one scene), as shown in, for example, FIG. 5 explained inthe first embodiment.

In the second automated driving mode, the ECU 20 time-serially predictsthe moving locus of the object for a predetermined time TS from time(for example, time T0) when the scene image is acquired. The movinglocus is predicted by predicting the position of the object for thepredetermined time TS at each time t1, t2, t3, . . . obtained bydividing the predetermined time TS.

The ECU 20 time-serially calculates the moving locus of the vehicle 1(self-vehicle) for the predetermined time TS at each time t1, t2, t3, .. . based on the moving locus of the object for the predetermined time.By a series of scene understanding, prediction, and action planningprocesses, the calculation result of the moving locus of the vehicle 1(self-vehicle) based on the calculation result of the moving locus ofthe object and the prediction result of the moving locus of the objectis obtained. The predetermined time TS is an arbitrarily settable time,and a shorter predetermined time TS can be set.

By performing a loop operation, the ECU 20 predicts the motion of theobject if the vehicle 1 (self-vehicle) moves in accordance with thecalculated moving locus. The moving locus of the object is predicted bypredicting the position of the object for the predetermined time TS ateach time t1, t2, t3, . . . obtained by dividing the predetermined timeTS.

The ECU 20 compares the predicted moving locus of the object with themoving locus of the object obtained by the last prediction processing.If, as a result of the comparison processing, the moving locus of theobject has changed, the ECU 20 recalculates (corrects) the moving locusof the vehicle 1 (self-vehicle) for the predetermined time TS at timet1, t2, t3, . . . based on the changed moving locus of the object.

If the same processes are performed a predetermined number of times bythe loop operation, and the moving locus of the object remainsunchanged, that is, a change in moving locus of the object hasconverged, the ECU 20 determines the moving locus of the vehicle 1(self-vehicle) based on the converged moving locus of the object.

If the change in moving locus has not converged even after the loopoperation is performed the predetermined number of times, the ECU 20recalculates (corrects) the moving locus of the vehicle 1 (self-vehicle)for the predetermined time TS at each time t1, t2, t3, . . . based onthe last corrected moving locus of the object, and then determines themoving locus of the vehicle 1 (self-vehicle). Note that the aboveprocessing has been described for explaining the operation associatedwith the scene understanding, prediction, and action planning processesin the scene image at time T0. The same applies to a case at time T1after time T0 or a case at time T2 after time T1.

FIG. 8 is a flowchart for explaining the procedure of processing in thesecond automated driving mode. When, for example, a driver instructssetting of a destination and automated driving, the ECU 20 automaticallycontrols traveling of the vehicle 1 to the destination in accordancewith a guidance route found by an ECU 24.

At the time of automated driving control, the ECU 20 acquiresinformation (external information) concerning the peripheral status ofthe vehicle 1 from ECUs 22 and 23 (step S20). By performing theprocessing in step S20, an image (scene image) representing theperipheral status of the vehicle is acquired based on the externalinformation at time T0, as shown in FIG. 5 explained in the firstembodiment.

The ECU 20 extracts objects (a pedestrian 204, the other vehicle 205,and an obstacle 217) included in the scene image at time (for example,time T0) when the external information is acquired (step S21). The ECU20 extracts, from the scene image, a static object (stationary object)and a dynamic object (moving object), and specifies the arrangement ofthe extracted objects in the scene image. The ECU 20 fixes the positionof the vehicle 1 (self-vehicle) in the scene image as the origin of acoordinate system, specifying the positions of the objects.

With respect to the dynamic object (moving object) among the extractedobjects, the ECU 20 predicts the motion (moving locus) of the object forthe predetermined time TS from time to, when the external information isacquired, toward the future (step S22). The ECU 20 time-seriallypredicts the position of the object for the predetermined time TS attime t1, t2, t3, . . . obtained by dividing the predetermined time TS.

Based on the time-serially calculated moving locus of the object for thepredetermined time, the ECU 20 calculates the moving locus of thevehicle 1 (self-vehicle) for the predetermined time (step S23). Based onthe prediction result of the moving locus of the object in the sceneimage acquired at time T0, the ECU 20 calculates the time-series movinglocus of the vehicle 1 (self-vehicle) in consideration of traveling databy a skilled driver and the positions of the objects existing around thevehicle 1 (self-vehicle) in the scene image. The ECU 20 time-seriallycalculates the position of the vehicle 1 (self-vehicle) for thepredetermined time TS at each time t1, t2, t3, . . . .

In step S24, if the vehicle 1 (self-vehicle) moves in accordance withthe moving locus calculated in step S23, the ECU 20 predicts the motionof the object. The ECU 20 compares the moving locus of the objectpredicted in this step with the moving locus of the object predicted instep S22 (the moving locus of the object obtained by the last predictionprocessing). The motion of the vehicle 1 (self-vehicle) may influencethe motion of the object, and the ECU 20 determines whether the motionof the object (the moving locus of the object) changes by receiving theinfluence of the motion (the moving locus of the self-vehicle) of thevehicle 1 (self-vehicle) for the predetermined time (step S24). Based oncomparison with a threshold, the ECU 20 determines whether a change incorrected moving locus of the object has converged by performing therepetitive operation in the predetermined operation period. Thepredetermined threshold is an arbitrarily settable value.

If it is determined in step S24 that a difference in moving locus of theobject exceeds the predetermined threshold, the ECU 20 determines thatthe moving locus of the object has changed (YES in step S24), andreturns the process to step S23.

In step S23, the ECU 20 recalculates (corrects) the moving locus of thevehicle 1 (self-vehicle) based on the changed moving locus of theobject. That is, if the moving locus of the object has changed, the ECU20 recalculates (corrects) the moving locus of the vehicle 1(self-vehicle) for the predetermined time TS at each time t1, t2, t3, .. . based on the changed moving locus of the object, and advances theprocess to step S24.

In step S24, if the vehicle 1 (self-vehicle) moves in accordance withthe moving locus corrected in step S23, the ECU 20 predicts again themotion of the object. Then, the ECU 20 compares the moving locus of theobject predicted again with the moving locus of the object obtained bythe last prediction processing. If, as a result of the comparisonprocessing, the difference in moving locus of the object exceeds thepredetermined threshold, the ECU 20 determines that the moving locus ofthe object has changed (YES in step S24), and returns the process tostep S23. In step S23, the ECU 20 time-serially recalculates (corrects)the moving locus of the vehicle 1 (self-vehicle) for the predeterminedtime TS at each time t1, t2, t3, . . . . The ECU 20 repeats the sameloop operation a preset number of times.

On the other hand, if, as a result of the comparison processing in stepS24, there is no difference in moving locus of the object (the movinglocus remains the same) or the difference in moving locus of the objectis equal to or smaller than the predetermined threshold, the ECU 20determines that the moving locus of the object remains unchanged (NO instep S24), and advances the process to step S25.

In step S25, the ECU 20 determines the moving locus along which thevehicle 1 (self-vehicle) moves for the predetermined time. If, based onthe determination processing in step S24, the change in corrected movinglocus of the object has converged (NO in step S24), the ECU 20determines the moving locus of the vehicle 1 (self-vehicle) based on theconverged moving locus of the object (step S25).

Note that if the ECU 20 counts up the number of times of execution ofthe loop operation in steps S23 and S24, and the change in moving locusdoes not converge even after the loop operation is performed thepredetermined number of times (YES in step S24), that is, the movinglocus of the object has changed to exceed the predetermined threshold,the ECU 20 advances the process to step S25 without returning theprocess to step S23.

If, based on the result of the comparison processing in step S24, thechange in corrected moving locus of the object does not converge evenafter the loop operation is performed the predetermined number of times(YES in step S24), the ECU 20 determines the moving locus of the vehicle1 (self-vehicle) based on the last corrected moving locus of the object(step S25).

Detailed contents of the processing in the second automated driving modewill be described next with reference to FIGS. 5 and 6A to 6C. In thesecond automated driving mode as well, an image (scene image)representing the peripheral status of the vehicle is acquired, as shownin FIG. 5 explained in the first embodiment. For example, at time T0, ascene image of objects existing around the traveling vehicle 1(self-vehicle), that has been acquired by the detection units (camera41, lidar 42, and radar 43), is acquired. Then, the ECU 20 extractsobjects (the pedestrian 204, the other vehicle 205, and the obstacle217) included in the scene image by the scene understanding processing,fixes the position of the vehicle 1 (self-vehicle) in the scene image asthe origin of a coordinate system, and thus specifies the positions ofthe objects. By performing the series of scene understanding,prediction, and action planning processes by one sequence, similarly tothe processes explained in the first embodiment, it is possible toobtain a moving locus 505 of the other vehicle 205 and a moving locus501 of the vehicle 1 (self-vehicle), as shown in FIG. 5.

In the second automated driving mode, if the vehicle 1 (self-vehicle)moves in accordance with the calculated moving locus 501, the ECU 20predicts the motion of each object by performing a loop operation. Themoving locus of the object is predicted by predicting the position ofthe object for the predetermined time TS at each time t1, t2, t3, . . .obtained by dividing the predetermined time TS.

If the vehicle 1 (self-vehicle) selects the moving locus 501 closer to acenter line 203 by avoiding the pedestrian 204, the other vehicle 205shifts (corrects) the moving locus toward a division line 201 (towardthe obstacle 217 to which the other vehicle 205 has a sufficientdistance) to avoid interference with the vehicle 1 (self-vehicle).

For example, if the moving locus 505 of the other vehicle 205 changes,by the loop operation, to a moving locus 506 indicated by a one-dotdashed line when the vehicle 1 (self-vehicle) moves along the movinglocus 501, the ECU 20 recalculates (corrects) the moving locus of thevehicle 1 (self-vehicle) for the predetermined time TS at each time t1,t2, t3, . . . based on the changed moving locus 506 of the other vehicle205. For example, by re-executing the calculation processing, the ECU 20corrects the moving locus 501 to a moving locus 502 indicated by aone-dot dashed line.

Similarly, if the moving locus 506 of the other vehicle 205 changes, bythe loop operation, to a moving locus 507 indicated by a solid line whenthe vehicle 1 (self-vehicle) moves along the moving locus 502, the ECU20 recalculates (corrects) the moving locus of the vehicle 1(self-vehicle) for the predetermined time TS at each time t1, t2, t3, .. . based on the changed moving locus 507 of the other vehicle 205. Forexample, by re-executing the calculation processing, the ECU 20 correctsthe moving locus 502 to a moving locus 503 indicated by a solid line.This processing corresponds to YES in step S24 of FIG. 8 and theprocessing in step S23.

If the moving locus 507 of the object (the other vehicle 205) remainsunchanged by the loop operation, the ECU 20 determines that the changein moving locus of the other vehicle 205 has converged, and determinesthe moving locus 503 shown in FIG. 5 as the moving locus of the vehicle1 (self-vehicle) based on the converged moving locus 507 of the othervehicle 205. These processes correspond to NO in step S24 of FIG. 8 andthe processing in step S25.

Furthermore, if the same processes are performed the predeterminednumber of times by the loop operation, and the change in moving locusdoes not converge even after the loop operation is performed thepredetermined number of times, the ECU 20 determines the moving locus ofthe vehicle 1 (self-vehicle) based on the last corrected moving locus ofthe object. These processes correspond to the processing (YES in stepS24 of FIG. 7 and step S25) executed when the change in moving locusdoes not converge even after the loop operation is performed thepredetermined number of times.

FIGS. 6A to 6C are views each exemplarily showing determination of themoving locus of the self-vehicle. Referring to FIG. 6B, each of curves601 and 602 indicates the distribution of prediction values at which theobject is predicted to exist at a predetermined position (a positionbetween P and P′ in FIG. 5) in the scene image, similarly to FIG. 6A.The curve 601 indicates the distribution of prediction values concerningthe pedestrian 204 shown in FIG. 5. The curve 602 indicates thedistribution of prediction values concerning the other vehicle 205 shownin FIG. 5.

In FIG. 6B, a broken-line curve 602B indicates a change in distributionof prediction values when the motion of the other vehicle 205 changes byreceiving the influence of the moving loci 501 to 503 (FIG. 5) of thevehicle 1 (self-vehicle). The distribution of the broken-line curve 602Bshifts toward a rectangle 604 (the right side on the drawing), ascompared to the distribution of the curve 602. The shift of thebroken-line curve 602B corresponds to the change from the moving locus505 to the moving locus 507 of the other vehicle 205 in FIG. 5.

A region between the curve 601 and the broken-line curve 602B is aregion where no object exists on the lane on which the vehicle 1(self-vehicle) travels, and a region where the vehicle 1 (self-vehicle)can travel. The travelable region in FIG. 6B is wider than that in FIG.6A due to the shift of the broken-line curve 602B. A curve 603Bindicates the distribution of traveling patterns by the skilled driverin the travelable region. A peak M2 of the curve 603B indicates atraveling position that is selected by the skilled driver at the highestprobability in the travelable region.

The ECU 20 specifies a region (travelable region) where no object existsat each time (for example, t1, t2, t3, . . . , TS) shown in FIG. 7, andcalculates, based on the distribution of the traveling patterns by theskilled driver, a moving locus so as to move the vehicle 1 to theposition of the peak M2 of the curve 603B. The thus calculated movinglocus is set as the moving locus 503 of the vehicle 1 (self-vehicle)obtained in steps S13 and S14 of FIG. 4.

In the second automated driving mode, the moving locus of the vehicle 1(self-vehicle) is calculated by predicting the motion of the object. Themoving locus of the object is corrected by predicting the motion of theobject in accordance with the moving locus of the vehicle 1(self-vehicle). Based on the corrected moving locus of the object, themoving locus of the vehicle 1 (self-vehicle) is corrected. By correctingthe moving locus of the vehicle 1 (self-vehicle) based on the change inmoving locus of the object by the loop operation, it is possible tocalculate the moving locus of the vehicle 1 (self-vehicle) at higheraccuracy in the action planning processing.

Therefore, since it is possible to determine the moving locus of theself-vehicle in consideration of the change in moving locus of theobject while considering accumulation of traveling data by the skilleddriver, it is possible to determine the moving locus of the self-vehiclefor the predetermined time at higher accuracy. In addition, since themoving locus is determined based on an action actually taken by theskilled driver, the self-vehicle takes an action that would be selectedby the skilled driver or an action close to the action in considerationof the peripheral environment. As a result, even in an environment suchas an urban area including many moving objects, it is possible todetermine the moving locus of the vehicle 1 (self-vehicle) in accordancewith the motions of traffic participants such as a pedestrian andanother vehicle.

Third Embodiment: Third Automated Driving Mode

In this embodiment as well, an ECU 20 executes control associated withautomated driving of a vehicle 1. The ECU 20 controls automated drivingof the vehicle 1 as a control target based on a calculated moving locus.In the third automated driving mode, the moving locus of an objectobtained in a scene image at given time T1 is corrected based on anaction plan (the moving locus of the vehicle 1 (self-vehicle)) obtainedin a scene image at time T0 before time T1, thereby calculating themoving locus of the vehicle 1 (self-vehicle) based on the correctedmoving locus of the object. The action plan calculated based on thescene image at time T0 is corrected by an action plan calculated basedon the scene image at time T1. That is, the moving locus of the vehicle1 (self-vehicle) calculated at time T0 is replaced by the moving locusof the vehicle 1 (self-vehicle) calculated at time T1.

FIG. 2C is a view showing an outline of a processing sequencecorresponding to the third automated driving mode. Processing at time T0is the same as the processing shown in FIG. 2A, in which the ECU 20performs scene understanding, prediction, and action planning processesby one sequence. The calculation result of the action plan (the movinglocus of the vehicle 1 (self-vehicle)) calculated at time T0 isreflected on processing at time T1.

In the processing at time T1, scene understanding, prediction, andaction planning processes are performed by one sequence. In theprediction processing, however, the calculation result of the actionplan at time T0 before time T1 is input.

The ECU 20 corrects the moving locus of the object calculated based onthe scene image acquired at time T1 using the action plan calculated attime T0, thereby time-serially calculating the moving locus of thevehicle 1 (self-vehicle) corresponding to the corrected moving locus ofthe object. The calculation result of the action plan (the moving locusof the vehicle 1 (self-vehicle)) calculated at time T1 is reflected onprocessing at time T2 in the same manner.

FIG. 9 is a view schematically showing the relationship between a timeaxis (ordinate) representing actual time when the vehicle 1(self-vehicle) travels and a calculation time axis (abscissa) to be usedfor calculation when obtaining the moving loci of a dynamic object(moving object) and the vehicle 1 (self-vehicle). For example, at timeT1, based on information (external information) acquired by detectionunits (camera 41, lidar 42, and radar 43), pieces of information of astatic object and dynamic object existing around the vehicle 1(self-vehicle) are obtained as an image representing the peripheralstatus of the vehicle (a scene image representing one scene), as shownin, for example, FIG. 11.

In the third automated driving mode, the ECU 20 time-serially calculatesthe moving locus of the object for a predetermined time TS from time(for example, time T1) when the scene image is acquired, andtime-serially calculates the moving locus of the vehicle 1(self-vehicle) based on the position of the moving locus of the objectat each time.

The ECU 20 predicts the moving locus along which the dynamic object suchas a pedestrian or another vehicle that moves with the lapse of timemoves for the predetermined time TS from time T1 when the scene image isacquired. In the prediction processing, the calculation result of theaction plan (the moving locus of the vehicle 1 (self-vehicle)) obtainedin the scene image at time T0 before time T1 is input.

The moving locus of the object is predicted by time-serially predictingthe position of the object for the predetermined time TS at each timet1, t2, t3, . . . obtained by dividing the predetermined time TS.

The ECU 20 corrects the prediction result of the moving locus of theobject based on the calculation result of the action plan (the movinglocus of the vehicle 1 (self-vehicle)) obtained in the scene image attime T0. The ECU 20 time-serially calculates the moving locus of thevehicle 1 (self-vehicle) for the predetermined time TS at each time t1,t2, t3, . . . based on the corrected moving locus of the object.

Based on the calculation result of the moving locus of the vehicle 1(self-vehicle) calculated at time T1, the ECU 20 corrects (updates) themoving locus of the vehicle 1 (self-vehicle) calculated at time T0. TheECU 20 reflects the calculation result of the action plan (the movinglocus of the vehicle 1 (self-vehicle)) calculated at time T1 on theprocessing at time T2 in the same manner.

FIG. 10 is a flowchart for explaining the procedure of processing in thethird automated driving mode. When, for example, a driver instructssetting of a destination and automated driving, the ECU 20 automaticallycontrols traveling of the vehicle 1 to the destination in accordancewith a guidance route found by an ECU 24.

At the time of automated driving control, the ECU 20 acquiresinformation (external information) concerning the peripheral status ofthe vehicle 1 from ECUs 22 and 23 (step S30). By performing theprocessing in step S30, a scene image shown in, for example, FIG. 11 isacquired based on the external information at time T1.

The ECU 20 extracts objects included in the scene image at time (forexample, time T1) when the external information is acquired (step S31).The ECU 20 extracts, from the scene image, a static object (stationaryobject) that does not move with the lapse of time, and a dynamic object(moving object) that moves with the lapse of time, and specifies thearrangement of the extracted objects in the scene image. The ECU 20fixes the position of the vehicle 1 (self-vehicle) in the scene image asthe origin of a coordinate system, specifying the positions of theobjects.

The ECU 20 inputs the calculated action plan (the calculation result ofthe moving locus of the self-vehicle) at previous time T0 (step S32).That is, the ECU 20 acquires the action plan (the moving locus of theself-vehicle) obtained in the previous scene image at time T0 beforetime T1 when the scene image is acquired.

With respect to the dynamic object (moving object) among the extractedobjects, the ECU 20 time-serially calculates the motion (moving locus)of the object for the predetermined time TS from time T1, when theexternal information is acquired, toward the future (step S33).

The ECU 20 corrects the moving locus of the object calculated in stepS33 based on the action plan (the moving locus of the self-vehicle)obtained in the previous scene image at time T0 and acquired in stepS32. Based on the moving locus of the vehicle 1 calculated in theprevious scene image acquired at time (for example, T0) before time (forexample, T1) when the scene image is acquired, the ECU 20 calculates themoving locus by correcting the moving locus of the object in the sceneimage. The motion of the vehicle 1 (self-vehicle) may influence themotion of the object, and the ECU 20 calculates a change in motion ofthe object (moving locus of the object) caused by the influence of themoving locus of the vehicle 1 (self-vehicle), and corrects the movinglocus of the object based on the calculation result of the change inmoving locus. The ECU 20 time-serially corrects the moving locus of theobject for the predetermined time TS at each time t1, t2, t3, . . .obtained by dividing the predetermined time TS.

In step S34, the ECU 20 time-serially calculates the moving locus of thevehicle 1 (self-vehicle) for the predetermined time based on thecorrected moving locus of the object. The ECU 20 time-seriallycalculates the moving locus of the vehicle 1 (self-vehicle) for thepredetermined time TS at each time t1, t2, t3, . . . obtained bydividing the predetermined time TS based on the corrected moving locusof the object in consideration of, for example, traveling data by askilled driver and the position of the object existing around thevehicle 1 (self-vehicle) in the scene image.

In step S35, based on the calculation result in step S34, the ECU 20corrects (updates) the action plan (the moving locus of the vehicle 1(self-vehicle)) calculated at time T0.

Detailed contents of the processing in the third automated driving modewill be described next with reference to FIG. 11. At time T1, a sceneimage of objects existing around the traveling vehicle 1 (self-vehicle),that has been acquired by the detection units (camera 41, lidar 42, andradar 43), is acquired (FIG. 11). The objects (a pedestrian 204, anothervehicle 205, and an obstacle 217) in FIG. 11 are the same as in FIG. 5.

The ECU 20 extracts the objects (the pedestrian 204, the other vehicle205, and the obstacle 217) included in the scene image shown in FIG. 11by the scene understanding processing, fixes the position of the vehicle1 (self-vehicle) in the scene image as the origin of a coordinatesystem, and thus specifies the positions of the objects. The aboveprocessing corresponds to steps S30 and S31 of FIG. 10.

Referring to FIG. 11, a moving locus 1101 indicated by a two-dot dashedline represents the action plan (the moving locus of the vehicle 1(self-vehicle)) calculated based on the scene image at time T0 beforetime T1. By performing the series of scene understanding, prediction,and action planning processes by one sequence, similarly to theprocesses explained in the first embodiment, it is possible to obtainthe moving locus 1101 of the vehicle 1 (self-vehicle). This processingcorresponds to step S32 of FIG. 10, in which the ECU 20 acquires thecalculated action plan (the calculation result of the moving locus ofthe self-vehicle) at previous time T0. This processing corresponds tostep S32 of FIG. 10.

A moving locus 1105 indicated by a two-dot dashed line represents theprediction result of the calculated moving locus of the other vehicle205 in the scene image at time T1. The ECU 20 calculates a change inmoving locus of the other vehicle 205 when the vehicle 1 (self-vehicle)moves along the moving locus 1101. The ECU 20 corrects the moving locusof the other vehicle 205 based on the calculation result of the changein moving locus. Referring to FIG. 11, a moving locus 1106 indicated bya solid line represents the corrected moving locus of the other vehicle205. Based on the moving locus (for example, 1101) of the vehiclecalculated in the previous scene image acquired at previous time (forexample, T0) before time (for example, T1) when the scene image isacquired, the ECU 20 calculates the moving locus (for example, 1106) bycorrecting the moving locus (for example, 1105) of the object in thescene image. The processing of obtaining the moving loci 1105 and 1106of the other vehicle 205 corresponds to step S33 of FIG. 10.

Next, the ECU 20 time-serially calculates the moving locus of thevehicle 1 (self-vehicle) for the predetermined time based on thecorrected moving locus 1106 of the other vehicle 205. Referring to FIG.11, a moving locus 1102 indicated by a solid line represents the movinglocus of the vehicle 1 (self-vehicle) calculated based on the correctedmoving locus 1106 of the other vehicle 205. The processing of obtainingthe moving locus 1102 corresponds to step S34 of FIG. 10.

Based on the calculated moving locus 1102 of the vehicle 1(self-vehicle), the ECU 20 corrects the action plan (the moving locus1101 of the vehicle 1 (self-vehicle)) calculated at time T0. Theprocessing of correcting the moving locus 1101 by the moving locus 1102corresponds to step S35 of FIG. 10.

FIGS. 6A to 6C are views each exemplarily showing determination of themoving locus of the self-vehicle. Referring to FIG. 6C, each of curves601 and 602 indicates the distribution of prediction values at which theobject is predicted to exist at a predetermined position (a positionbetween P and P′ in FIG. 11) in the scene image. The curve 601 indicatesthe distribution of prediction values concerning the pedestrian 204shown in FIG. 11. The curve 602 indicates the distribution of predictionvalues concerning the other vehicle 205 shown in FIG. 11.

In FIG. 6C, a broken-line curve 602C indicates a change in distributionof prediction values when the motion of the other vehicle 205 changes byreceiving the influence of the moving locus 1101 (FIG. 11) of thevehicle 1 (self-vehicle). The distribution of the broken-line curve 602Cshifts toward a rectangle 604 (the right side on the drawing), ascompared to the distribution of the curve 602. The shift of thebroken-line curve 602C corresponds to the change from the moving locus1105 to the moving locus 1106 of the other vehicle 205 in FIG. 11.

A region between the curve 601 and the broken-line curve 602C is aregion where no object exists on the lane on which the vehicle 1(self-vehicle) travels, and a region where the vehicle 1 (self-vehicle)can travel. The travelable region in FIG. 6C is wider than, for example,the travelable region in FIG. 6A due to the shift of the broken-linecurve 602C. A curve 603C indicates the distribution of travelingpatterns by the skilled driver in the travelable region. A peak M3 ofthe curve 603C indicates a traveling position that is selected by theskilled driver at the highest probability in the travelable region. Acurve 603 indicates the distribution traveling patterns by the skilleddriver in the scene image at time T0, and corresponds to the curve 603shown in FIG. 6A.

The ECU 20 specifies a region (travelable region) where no object existsat each time (for example, t1, t2, t3, . . . , TS) shown in FIG. 9, andcalculates, based on the distribution of the traveling patterns by theskilled driver, a moving locus so as to move the vehicle 1 to theposition of the peak M3 of the curve 603C. The thus calculated movinglocus is set as the moving locus 1102 of the vehicle 1 (self-vehicle)obtained in step S34 of FIG. 10. In FIG. 6C, a position M1 is on themoving locus of the vehicle 1 calculated based on the scene image attime T0, and is corrected to M3 by correcting the moving locus of thevehicle 1 (self-vehicle).

In the third automated driving mode, the moving locus of the objectexisting around the vehicle 1 in the scene image acquired at time T1 iscorrected based on the calculation result of the action plan (the movinglocus of the vehicle 1 (self-vehicle)) calculated at previous time T0.It is possible to reduce the load of the operation processing bycorrecting the moving locus of the object based on the moving locus ofthe vehicle calculated in the previous scene image acquired at timebefore time when the scene image is acquired. This makes it possible tocorrect the moving locus of the object while reducing the calculationload, as compared to the loop operation of repeating the prediction andaction planning processes, and time-serially calculate the moving locusof the vehicle 1 (self-vehicle) based on the corrected moving locus ofthe object.

Other Embodiments

Several preferred embodiments have been described above. However, thepresent invention is not limited to these examples and may partially bemodified without departing from the scope of the invention. For example,another element may be combined with the contents of each embodiment inaccordance with the object, application purpose, and the like. Part ofthe contents of a certain embodiment may be combined with the contentsof another embodiment. In addition, individual terms described in thisspecification are merely used for the purpose of explaining the presentinvention, and the present invention is not limited to the strictmeanings of the terms and can also incorporate their equivalents.

Furthermore, a program that implements at least one function describedin each embodiment is supplied to a system or an apparatus via a networkor a storage medium, and at least one processor in the computer of thesystem or the apparatus can read out and execute the program. Thepresent invention can be implemented by this form as well.

Summary of Embodiments

Arrangement 1. A vehicle control apparatus according to the aboveembodiments is a vehicle control apparatus (for example, 100) thatcontrols automated driving of a vehicle (for example, 1) based on agenerated moving locus, comprising:

an extraction unit (for example, 20, 22, 23) configured to extract anobject (for example, 204, 205) existing around the vehicle (1) from ascene image (for example, FIG. 5) representing a peripheral status ofthe vehicle (1); and a control unit (for example, 20) configured tocalculate a moving locus (for example, 505) of the object and a movinglocus (for example, 501) of the vehicle for a predetermined time (forexample, TS) from time (for example, T0) when the scene image isacquired, and generate a moving locus (for example, 506, 507) bycorrecting the moving locus of the object based on the moving locus(501) of the vehicle.

According to arrangement 1, it is possible to obtain the moving locus ofthe object by correcting a predicted motion based on the moving locus ofthe vehicle as a control target.

Arrangement 2. In the vehicle control apparatus (100) according to theabove embodiment, the control unit (20) generates a moving locus (forexample, 502, 503) by correcting the moving locus of the vehicle basedon the corrected moving locus (506, 507) of the object.

According to arrangement 2, it is possible to obtain the correctedmoving locus of the vehicle based on the corrected moving locus of theobject.

Arrangement 3. In the vehicle control apparatus (100) according to theabove embodiment, the control unit (20) performs repetitive operationsfor the calculation of the moving locus and the correction of the movinglocus in a predetermined period.

Arrangement 4. In the vehicle control apparatus (100) according to theabove embodiment, the control unit (20) determines, based on comparisonwith a threshold, whether a change in the corrected moving locus of theobject has converged by the repetitive operations in the period.

Arrangement 5. In the vehicle control apparatus (100) according to theabove embodiment, if, based on the determination, the change in thecorrected moving locus of the object has converged, the control unit(20) generates the moving locus (for example, 503 in FIG. 5) bycorrecting the moving locus of the vehicle based on the converged movinglocus (for example, 507 in FIG. 5) of the object.

Arrangement 6. In the vehicle control apparatus (100) according to theabove embodiment, if, based on the determination, the change in thecorrected moving locus of the object has not converged in the period,the control unit (20) generates the moving locus of the vehicle based ona last corrected moving locus of the object.

According to arrangements 3 to 6, it is possible to calculate the movinglocus of the object and that of the vehicle 1 at high accuracy byperforming the repetitive operation in the predetermined period.

Arrangement 7. In the vehicle control apparatus (100) according to theabove embodiment, by performing image processing for the scene image,the extraction unit (20, 22, 23) extracts, from the scene image, astatic object (for example, 217 in FIG. 5) that does not move with alapse of time and a dynamic object (for example, 205 in FIG. 5) thatmoves with the lapse of time.

According to arrangement 7, it is possible to specify, among the objectsincluded in the scene image, an object whose moving locus is to beobtained.

Arrangement 8. In the vehicle control apparatus (100) according to theabove embodiment, the control unit (20) executes operation processing byfixing a position of the vehicle in the scene image as an origin of acoordinate system.

According to arrangement 8, it is possible to reduce the load of theoperation processing in the calculation of the moving locus by fixing,as the origin of the coordinate system, the position of the vehicle asthe control target in the image acquired at given time.

Arrangement 9. A vehicle (for example, 1) according to the aboveembodiment comprises a vehicle control apparatus (for example, 100)defined in any one of arrangements 1 to 8 described above.

According to arrangement 9, it is possible to control automated drivingof the vehicle based on the moving locus generated at higher accuracy,by executing the above-described processing in the vehicle quickly.

Arrangement 10. A vehicle control method according to the aboveembodiments is a vehicle control method for a vehicle control apparatusthat controls automated driving of a vehicle based on a generated movinglocus, comprising:

an extraction step (for example, S20, S21) of extracting an objectexisting around the vehicle from a scene image (for example, FIG. 5)representing a peripheral status of the vehicle; and

a control step (for example, S22-S24) of performing operations forcalculating a moving locus (for example, 505) of the object and a movinglocus (for example, 501) of the vehicle for a predetermined time (forexample, TS) from time (for example, T0) when the scene image isacquired, and generating a moving locus (for example, 506, 507) bycorrecting the moving locus of the object based on the moving locus(501) of the vehicle.

According to arrangement 10, it is possible to obtain the moving locusof the object by correcting a predicted motion based on the moving locusof the self-vehicle.

Arrangement 11. In the vehicle control method according to the aboveembodiments, in the control step (for example, S22-S24), a moving locus(for example, 502, 503) is generated by correcting the moving locus ofthe vehicle based on the corrected moving locus (506, 507) of theobject.

According to arrangement 11, it is possible to obtain the correctedmoving locus of the vehicle based on the corrected moving locus of theobject.

Arrangement 12. In the vehicle control method according to the aboveembodiments, in the control step (for example, S22-S24), the operationsfor the calculation of the moving locus and the correction of the movinglocus are performed in a predetermined period.

According to arrangement 12, it is possible to calculate the movinglocus of the object and that of the vehicle 1 at high accuracy byperforming the repetitive operation in the predetermined period.

Arrangement 13. A storage medium according to the above embodimentsstores a program for causing a computer to execute each step of avehicle control method defined in any one of arrangements 10 to 12described above. According to arrangement 13, it is possible toimplement each step of the vehicle control method by the computer.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2017-166064, filed Aug. 30, 2017, which is hereby incorporated byreference wherein in its entirety.

What is claimed is:
 1. A vehicle control apparatus that controlsautomated driving of a vehicle based on a generated moving locus,comprising: an extraction unit configured to extract an object existingaround the vehicle from a scene image representing a peripheral statusof the vehicle; and a control unit configured to calculate a movinglocus of the object and a moving locus of the vehicle for apredetermined time from time when the scene image is acquired, andgenerate a moving locus by correcting the moving locus of the objectbased on the moving locus of the vehicle.
 2. The apparatus according toclaim 1, wherein the control unit generates a moving locus by correctingthe moving locus of the vehicle based on the corrected moving locus ofthe object.
 3. The apparatus according to claim 1, wherein the controlunit performs repetitive operations for the calculation of the movinglocus and the correction of the moving locus in a predetermined period.4. The apparatus according to claim 3, wherein the control unitdetermines, based on comparison with a threshold, whether a change inthe corrected moving locus of the object has converged by the repetitiveoperations in the period.
 5. The apparatus according to claim 4, whereinif, based on the determination, the change in the corrected moving locusof the object has converged, the control unit generates the moving locusby correcting the moving locus of the vehicle based on the convergedmoving locus of the object.
 6. The apparatus according to claim 4,wherein if, based on the determination, the change in the correctedmoving locus of the object has not converged in the period, the controlunit generates the moving locus of the vehicle based on a last correctedmoving locus of the object.
 7. The apparatus according to claim 1,wherein by performing image processing for the scene image, theextraction unit extracts, from the scene image, a static object thatdoes not move with a lapse of time and a dynamic object that moves withthe lapse of time.
 8. The apparatus according to claim 1, wherein thecontrol unit executes operation processing by fixing a position of thevehicle in the scene image as an origin of a coordinate system.
 9. Avehicle comprising a vehicle control apparatus defined in claim
 1. 10. Avehicle control method for a vehicle control apparatus that controlsautomated driving of a vehicle based on a generated moving locus,comprising: an extraction step of extracting an object existing aroundthe vehicle from a scene image representing a peripheral status of thevehicle; and a control step of performing operations for calculating amoving locus of the object and a moving locus of the vehicle for apredetermined time from time when the scene image is acquired, andgenerating a moving locus by correcting the moving locus of the objectbased on the moving locus of the vehicle.
 11. The method according toclaim 10, wherein in the control step, a moving locus is generated bycorrecting the moving locus of the vehicle based on the corrected movinglocus of the object.
 12. The method according to claim 10, wherein inthe control step, the operations for the calculation of the moving locusand the correction of the moving locus are performed in a predeterminedperiod.
 13. A storage medium storing a program for causing a computer toexecute each step of a vehicle control method defined in claim 10.